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Motivation  for  Project 

■  Effective  Government  decision-making  for  Environmental,  Safety 
and  Occupational  Health  (ESOH)  technology  Investments  and 
operations  requires  current  information  on  many  issues,  including: 

-  Related  technology  developments  and  trends 

-  Active  organizations  /  individuals 

■  Massive  amounts  of  relevant  Information  exist  on  the  web,  but  are 
scattered  across  many  sources  and  are  often  not  readily  available 

-  Technology  press  releases  -  Organization  web  sites 

-  Technicai  and  trade  journais  -  Conference  proceedings 

■  Gathering  and  understanding  this  data  is  a  significant  challenge 
for  program  managers,  technology  developers,  technology 
purchasers,  and  other  decision-makers 
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The  Problem 

General  search  engines  index  only  the  “Surface  Web”  - 
only  reaching  16%  of  the  available  information 


16% 


84% 
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Categories  of  Web  Information 


SURFACE  WEB 


HIDDEN  WEB 


r 
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•  Free  Access 


•  Free  Access 

-  Indexed  by  search 
engines  such  as 
Google 


-  Not  indexed  by 
traditional  search 
engines 

-  Need  to  know  where  to 
look  and  how  to  search 


•  Restricted  Access 

-  Paid  Subscription  or 
Registration 

-  Not  indexed  by 
traditional  search 
engines 

-  Need  to  know  where  to 
look  and  how  to  search 
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Deep  Web  Examples 


Resource  Category 

Sampie  Sites 

Find 

Using 

Googie? 

Find  with 
Deep  Web 
Search? 

New  Product  Announcements  - 

Sites  that  publish  press  releases, 
product  announcements  or  featured 
products 

GlobalSpec  -  Product 
Announcements 

Reed  Business  Information 

No 

Yes 

Trade  Journals  -  Journals  devoted 
to  specific  applications,  typically 
include  product  directories,  product 
announcements,  articles  by  company 
representatives 

Pennwell  -  Water  & 

Wastewater  International 

Paint  &  Coatings  Industry 
Magazine 

No 

Yes 

Technical  Literature  -  Technical 
journals  containing  articles  from 
universities  and  companies 

SciRus 

SciTation 

Google  Scholar 

No 

Yes 

Product  directories 

Thomas  Register 

Global  Spec  -  product 
directory 

Limited 

Yes 

Funded  Research 

EPA  Funded  Research  Clu-ln 

No 

Yes 
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The  Benefits  of  a  Deep  Web  Search  Portal 

■  Opening  up  relevant  portions  of  the  hidden  Web  (84%  of  the  total 
web)  to  routine  searching  -  would  allow  a  consistent  set  of 
resources  to  be  available  to  all  users 

■  More  targeted  searching  of  the  best  sites  -  site  source  list  updated 
by  site  manager  so  new  sites  become  available  to  all  users  as  they 
are  identified 

■  Automatic  quick  searches  of  the  best  sites  for  priority  topics  and 
materials  needed  -  user  does  not  need  to  learn  the  search 
commands  for  several  different  sites 

■  Faster  searches  of  relevant  Web  sites  for  environmental 
technologies  -  cuts  wasted  time  finding  sites 

■  On-line  ability  to  organize  and  screen  post-search  results  -  cuts 
wasted  time  to  evaluate  initial  search  results 

■  Reduced  risk  of  missing  key  information  that  could  affect  future 
environmental  actions 
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Task  Objectives 

1 .  Develop  and  test  an  advanced,  strategic  data  search 
and  analysis  system  prototype  based  on  commercial-off- 
the-shelf  (COTS)  tools  for  online  searching 

2.  Identify  a  Department  of  Defense  (DoD)-wide  group  of 
potential  system  users  and  other  stakeholders  to  test  the 
prototype  design  and  support  operational 
implementation 

3.  Develop  the  business  model  for  operational  system 
deployment  and  the  roadmap  to  full  implementation  for 
DoD  users 
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Objective  1  Accomplishments- 
Prototype  System 

■  Completed  User  Needs  Assessment 

■  Completed  Prototype  System  Design 

-  Screened  34  COTS  deep  web  search  engines 

-  Evaluated  7  leading  COTS  deep  web  search  engines 

-  Selected  2  COTS  vendors  for  demonstration 

■  Bright  Planet  (http://www.brightplanetcom/) 

■  Deep  Web  Technologies  (http://www.deepwebtech.com/) 

■  Developed  Prototype  Advanced  Web  Search  Portal 

■  Completed  Proof  of  Concept  Demonstration/  Validation  for  the  two 
candidate  systems 

■  Identified  a  third  candidate  system  with  potentially  superior 
capabilities  at  reasonable  cost  —  Exalead  (http://www.exalead.com/search) 
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Deep  Web  Mining  Approaches 


APPROACHES 

HARVESTER 

FEDERATED 

CRAWLER 

DATA 

COLLECTION 

METHOD 

Collects  materials  from 

identified  sites  based  on 
defined  filter,  reindexes 
materials  for  additional 
search  capabilities 

Connections  are  configured 
for  defined  Web  sites, 
search  is  run  simultaneously 
on  all  sites 

1  ndexes  designated  sites, 
index  identifies  key 
concepts  and  incorporates 
additional  capabilities 

ADVANTAGES 

■  Focused  collection 

reduces  search  time 

■  Material  can  be 
organized  into  categories 
for  browsing 

■  Searches  are  not  restricted 
by  filter  concepts 

■  Limited  field  searching 
may  be  available 

■  Results  are  always  current 

■  Searches  are  not 
restricted  by  filter 
concepts 

■Taxonomy  or  search 
suggestions  available 

■  Provides  both  speed  and 
flexibility  of  search  topic 

DIS¬ 

ADVANTAGES 

Certain  sites,  such  as 
GlobalSpec,  cannot 
be  harvested 

Quality  of  search  results 
depend  on  quality  of 
each  site’s  search 
engine 

Proof -of -concept  not 
completed 

EXAMPLE 

SYSTEMS 

BrightPlanet 

Deep  Web  Technologies 

Exalead 
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Key  Features  of  the  Prototyped  Systems 

■  Design  Concept; 

-  “One-stop  shopping”  Web  portal  for  searching  pre-selected  online 
sites,  including  deep  Web  applications,  to  access  ESOH 
technology,  news  and  other  information 

-  Efficient,  useful  responses  to  straightforward  search  queries 

■  Key  Components: 

-  COTS  deep  Web  search  tools  (three  tools  selected  for  evaluation) 

-  Search  results  provided  as  highly  ranked  URL  locations 

-  Search  results  files  can  be  manipulated  for  screening  and 
organization  of  key  content 

■  Access:  Password-protected  access  for  authorized  DoD  users 

■  User  Interface:  User-fhendly  to  accommodate  novice  searchers 
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Objective  2  Accomplishments  - 
Project  Stakeholders 


Agency 

Organizations 

Individuals 

OSD 

5 

10 

Army 

7 

15 

Navy,  Marines 

8 

19 

Air  Force 

5 

11 

NASA 

1 

2 
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Objective  3  Accomplishments- 
Business  Model  (Still  Being  Refined) 

■  Use  DENIX  as  Search  Engine  Host,  User  Access,  &  Help  Desk 

■  License  Deep  Web  Search  Engine  via  DENIX  Program  Office 

■  Use  ESOH  Advisory  Groups  to  define  lists  of  primary  web  sites  for 
searches  to  keep  system  responsive  to  their  search  requirements: 

-  PAO  Review  -  Emerging  Contaminants  Tracking 

-  RDT&E  Investment  -  Equipment/Services  Procurement 

■  Contract  for  Paid  Subscription  Sites  via  DENIX  Program  Office 

■  Advisory  Group  &  Search  Engine  Support  provided  by  NDCEE 

■  Cost  Sharing  Between  DENIX  Program  Office  &  Primary  Users 

-  Target  Using  Organization  Cost  About  $50K 
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Next  Steps 

■  Plan  and  execute  final  COTS  system  proof-of-concept 
demonstration  (June  2007) 

■  Final  Design  Report  (July  2007) 

■  Technology  Transfer  Information  Report:  Business  model 
and  roadmap  to  operational  system  design  and  use 
(August  2007) 

■  Final  Task  Report  (September  2007) 
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